Clustering method with a two-stage local binary pattern and an iterative image testing system thereof

ABSTRACT

A clustering method with a two-stage local binary pattern includes generating a gradient direction value according to a center sub-block and neighbor sub-blocks of a patch of an image; quantizing the gradient direction value, thereby generating a quantized gradient direction value; generating a gradient magnitude value according to the gradient direction value; quantizing the gradient magnitude value, thereby generating a quantized gradient magnitude value; concatenating the quantized gradient direction value and the quantized gradient magnitude value to generate a two-stage local binary pattern (2SLBP) value; and performing clustering of super-resolution imaging by using the 2SLBP value as an index.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention generally relates to super-resolution (SR), andmore particularly to a clustering method with a two-stage local binarypattern (2SLBP) and an iterative image testing system thereof.

2. Description of Related Art

High-definition displays grow rapidly nowadays. However, there areabundant image capturing devices, for example, surveillance devices,that produce low-resolution images. To fill this gap, super-resolution(SR) techniques have been proposed. Example-based super resolution isone of the SR techniques that predicts a high-resolution (HR) image bysearching high-resolution patches in a patch database, and replaces acorresponding low-resolution (LR) sub-block of a low-resolution inputimage with a retrieved high-resolution patch.

In this specification, “high-resolution” and “low-resolution” are termsused relative to each other. Therefore, a “high-resolution” image refersto any suitable image having a higher resolution than an image referredto as a “low-resolution” image.

In order to reduce the size of the patch database, coefficients (ordescriptors) of a mapping function, instead of HR patches themselves,are commonly stored in the patch database by using regression technique.

Local binary pattern (LBP) is a type of visual descriptor that isconventionally used to describe local geometry of a patch in the patchdatabase in order to group together the patches that look the same orhaving similar visual properties. However, the conventional LBPdisadvantageously results in abundant rarely occurring clusters (e.g.,with usage less than 0.001% of total), therefore making redundancy forstorage. Therefore, a need has thus arisen to propose a novel method toimprove on clustering.

SUMMARY OF THE INVENTION

In view of the foregoing, it is an object of the embodiment of thepresent invention to provide a clustering method with a two-stage localbinary pattern (2SLBP) and an iterative image testing system thereofthat are capable of substantially reducing rarely occurring clusters,therefore preventing redundancy or wasting of storage and enhancingstrength of super-resolution (SR) imaging tasks.

According to one embodiment, an image is divided into a plurality ofpatches. A gradient direction value is generated according to a centersub-block and neighbor sub-blocks of the patch; and the gradientdirection value is quantized, thereby generating a quantized gradientdirection value. A gradient magnitude value is generated according tothe gradient direction value; and the gradient magnitude value isquantized, thereby generating a quantized gradient magnitude value. Thequantized gradient direction value and the quantized gradient magnitudevalue are concatenated to generate a two-stage local binary pattern(2SLBP) value, which is used as an index to perform clustering ofsuper-resolution imaging.

According to another embodiment, an iterative image testing system basedon a two-stage local binary pattern (2SLBP) for super-resolution imagingincludes an interpolation device that receives a low-resolution inputimage and accordingly generates an interpolated pixel within a patch ofthe input image; a clustering device for generating a two-stage localbinary pattern (2SLBP) value as an index; a mapping device that stores aplurality of function tables, each including plural mapping functions,one of which is retrieved according to the index; and a predictiondevice that maps the interpolated pixel to an enhanced pixel accordingto the retrieved mapping function, thereby generating a high-resolutionoutput image. Moreover, the patch with the enhanced pixel is fed back togo through the clustering device, the mapping device and the predictiondevice at least one time to generate an updated enhanced pixel,therefore refining the output image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a flow diagram illustrated of a clustering method with atwo-stage local binary pattern (2SLBP) according to one embodiment ofthe present invention;

FIG. 2A to FIG. 2C demonstrate generating mean differences betweenneighbor sub-blocks and a center sub-block;

FIG. 3A schematically shows exemplary mean values of the centersub-block and the neighbor sub-blocks;

FIG. 3B graphically shows mean differences associated with correspondinggradients;

FIG. 3C schematically shows quantized mean differences associated withcorresponding gradients; and

FIG. 4 shows a block diagram illustrated of an image testing systembased on a two-stage local binary pattern (2SLBP) for super-resolution(SR) imaging according to one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

FIG. 1 shows a flow diagram illustrated of a clustering method 100 witha two-stage local binary pattern (2SLBP) according to one embodiment ofthe present invention. The clustering method 100 may be adaptable toperforming imaging tasks (for example, training and testing) on imagesignals for super-resolution (SR) such as example-based superresolution. The steps of the clustering method 100 may be performed byan electronic circuit such as a digital image processor, and thesuper-resolution imaging task may be implemented by hardware, softwareor their combinations.

In step 11, an image subject to clustering is divided into patches witha predetermined size (e.g., 7×7). FIG. 2A shows an exemplary patch witha size of 7×7. FIG. 2A also shows a center sub-block 21 (e.g., with asize of 3×3) having a center pixel 211 located in the middle of thepatch.

In step 12, a gradient direction value is generated. In one embodiment,a mean difference value (abbreviated as “mean difference” hereinafter)between a neighbor sub-block and the center sub-block 21 is generated torepresent the direction of a gradient between the neighbor sub-block andthe center sub-block 21. Specifically, the mean of the neighborsub-block is computed, and the mean of the center sub-block 21 iscomputed. Subsequently, the mean difference is generated by subtractingthe mean of the center sub-block 21 from the mean of the neighborsub-block. In one embodiment, a weighted mean of the neighbor sub-blockand a weighted mean of the center sub-block are generated instead.

FIG. 2B further shows a neighbor sub-block 22 (with the same size of3×3) having a center pixel 221 located upper-middle to the centersub-block 21. It is noted that the mean difference between the neighborsub-block 22 and the center sub-block 21 is indicated by an arrow, whichrepresents the direction of a gradient between the two sub-blocks 22 and21.

Step 22 as described above is repeated for other directions (e.g.,upper-right, right, lower-right, lower-middle, lower-left, left andupper-left) until all directions or predetermined portion of alldirection have been exhausted (step 13). FIG. 2C further shows aneighbor sub-block 23 (with the same size of 3×3) having a center pixel231 located upper-right to the center sub-block 21. It is noted that themean difference between the neighbor sub-block 23 and the centersub-block 21 is indicated by an arrow, which represents the direction ofa gradient between the two sub-blocks 23 and 21.

In step 14, the mean differences generated from step 12 are quantized togenerate a local multi-gradient level pattern (LMGP) value (or aquantized gradient direction value). For example, mean differencesbetween eight neighbor sub-blocks and the center sub-block 21 arequantized. Accordingly, this gives an 8-digit number. In one exemplaryembodiment, the mean differences are quantized into one of threequantization levels 0, 1 or 2, therefore resulting in a ternary number.The generation of the mean differences and the quantization thereof maybe expressed as follows:

${{LMPG} = {\sum\limits_{k = 0}^{k = {D - 1}}{{f\left( \frac{{\sum\limits_{o = 1}^{o = 9}{w_{o} \times P_{o}}} - {\sum\limits_{c = 1}^{c = 9}{w_{c} \times P_{c}}}}{\sum\limits_{i = 1}^{i = 9}w_{i}} \right)} \cdot 3^{k}}}},{{f(x)} = \left\{ \begin{matrix}{2,} & {x > \theta} \\{1,} & {{- \theta} \leq x \leq \theta} \\{0,} & {x < {- \theta}}\end{matrix} \right.}$where w_(o), w_(c) and w_(i) are weightings, P_(o) is a pixel value ofthe neighbor sub-blocks 22/23, P_(c) is a pixel value of the centersub-block 21, and θ is a predetermined threshold.

FIG. 3A schematically shows exemplary mean values of the centersub-block 21 and the neighbor sub-blocks 22/23. FIG. 3B graphicallyshows mean differences associated with corresponding gradients withthreshold θ of 30. FIG. 3C schematically shows quantized meandifferences associated with corresponding gradients. The resultant LMGPvalue may be represented as 01200100 in ternary, which is equivalent todecimal 1224.

In step 15, a gradient magnitude value is generated to represent themagnitude of gradients between the neighbor sub-blocks 22/23 and thecenter sub-block 21. In one embodiment, a root mean square (RMS) of themean differences is generated. In other words, the square root of thearithmetic mean of the squares of the mean differences is generated.

In step 16, the root mean square (RMS) of the mean differences (i.e.,gradient magnitude value) generated from step 15 are quantized togenerate a quantized gradient magnitude value. In one exemplaryembodiment, the generation of the root mean square (RMS) of the meandifferences and the quantization thereof may be expressed as follows:

${C_{M} = {f\left( \sqrt{\frac{1}{8}{\sum\limits_{k = 0}^{7}\left( {p_{k} - p_{c}} \right)^{2}}} \right)}},{{f(x)} = \left\{ \begin{matrix}{{T - 1},} & {x > \Psi_{1}} \\{{T - 2},} & {\Psi_{2} < x \leq \Psi_{1}} \\\vdots & \; \\{0,} & {x \leq \Psi_{{T - 1}\;}}\end{matrix} \right.}$where p_(k) is a mean value of a neighbor sub-block 22/23, p_(c) is amean value of the center sub-block 21, and Ψ₁, Ψ₂, Ψ_(T−1) arepredetermined thresholds.

In step 17, the LMGP value (LMGP) generated in step 14 (i.e., the firststage) and the quantized gradient magnitude value (C_(M)) generated instep 16 (i.e., the second stage) are concatenated (or joined) togenerate a two-stage local binary pattern (2SLBP) value. In oneembodiment, the digits of the quantized gradient magnitude value (C_(M))are more significant than the digits of the LMGP value (LMGP). The 2SLBPvalue generated in step 17 may be used as an index in performingclustering (step 18) of the super-resolution (SR) imaging tasks (forexample, training and testing).

According to the clustering method 100 as described above, as thegradient direction and the gradient magnitude are separately generatedand quantized, the resultant index for performing clustering is morerobust than conventional counterparts. For example, rarely occurringclusters (e.g., with usage less than 0.001% of total) using theclustering method 100 of the present embodiment may be substantiallyreduced compared to the conventional counterparts, therefore preventingredundancy or wasting of storage and enhancing strength ofsuper-resolution (SR) imaging tasks.

FIG. 4 shows a block diagram illustrated of an iterative image testingsystem 400 based on a two-stage local binary pattern (2SLBP) forsuper-resolution (SR) imaging according to one embodiment of the presentinvention. The blocks of the image testing system 400 may be implementedby hardware, software or their combinations.

In the embodiment, the iterative image testing system 400 may include aninterpolation device 41 that is coupled to receive a low-resolution (LR)input image, and is configured to generate an interpolated pixel withina patch (e.g., with a size of 7×7). A suitable interpolation method,such as Bicubic interpolation, may be used in the interpolation device41. A patch with the interpolated pixel is then subject to clusteringbased on the two-stage local binary pattern (2SLBP) by a clusteringdevice 42 performing steps as illustrated in FIG. 1, thereforegenerating an index.

In one embodiment, the patch with the interpolated pixel is normalizedbefore the patch is subject to clustering by the clustering device 42.For example, luminance normalization is performed by subtracting a patchmean from pixel values of the patch.

The iterative image testing system 400 may further include a mappingdevice 43 that includes a plurality of function tables, each includingplural mapping functions that are commonly stored as matrix coefficientsin a memory device of the mapping device 43. It is noted that themapping functions are provided by an image training system (not shown)based on the two-stage local binary pattern (2SLBP). The image trainingsystem may be implemented by a conventional technique, which is thusomitted here for brevity. One mapping function among the providedmapping functions is then retrieved according to the index generated bythe clustering device 42.

The iterative image testing system 400 may further include a predictiondevice 44 that is configured to map the interpolated pixel (generated bythe interpolation device 41) to an enhanced pixel according to theretrieved mapping function (from the mapping device 43). Therefore, theenhanced pixel along with other pixels in the patch may form ahigh-resolution (HR) output image. It is noted that, according to theembodiment, the pixel subject to clustering is pre-interpolated (by theinterpolation device 41), and image quality of the interpolated pixel isthen enhanced (by the prediction device 44) rather than being scaled up.

If the patch is normalized (e.g., luminance normalization) before thepatch is subject to clustering, the patch of the HR output image shouldbe de-normalized (i.e., the reverse of normalization). For example, thepatch mean should be added to pixel values of the patch of the HR outputimage.

According to one aspect of the embodiment, the patch with the enhancedpixel may be repeatedly (or iteratively) subject to the image trainingsystem (not shown) to provide an updated function table including pluralmapping functions. As shown in FIG. 4, the patch with the enhanced pixelis then fed back to go through the clustering device 42, the mappingdevice 43 and the prediction device 44 to generate an updated enhancedpixel, therefore refining the high-resolution (HR) output image. Theiteration of the embodiment may be executed a predetermined number oftimes, each time with a refined updated function table. In practice, asthe image training system is commonly offline and the iterative imagetesting system 400 is ordinarily online, the plurality of updatedfunction tables may thus be stored in the memory device of the mappingdevice 43 as a whole, and each iteration may be executed with acorresponding updated function table.

Although specific embodiments have been illustrated and described, itwill be appreciated by those skilled in the art that variousmodifications may be made without departing from the scope of thepresent invention, which is intended to be limited solely by theappended claims.

What is claimed is:
 1. An iterative image testing system based on atwo-stage local binary pattern (2SLBP) for super-resolution imaging, thesystem comprising: an interpolation device that receives alow-resolution input image and accordingly generates an interpolatedpixel within a patch of the input image; a clustering device performingthe following steps: generating a plurality of gradient direction valuesaccording to a center sub-block and neighbor sub-blocks of the patch;quantizing the gradient direction values, thereby generating a quantizedgradient direction value; generating a gradient magnitude valueaccording to the gradient direction values; quantizing the gradientmagnitude value, thereby generating a quantized gradient magnitudevalue; and concatenating the quantized gradient direction value and thequantized gradient magnitude value to generate a two-stage local binarypattern (2SLBP) value as an index; a mapping device that stores aplurality of function tables, each including plural mapping functions,one of which is retrieved according to the index; and a predictiondevice that maps the interpolated pixel to an enhanced pixel accordingto the retrieved mapping function, thereby generating a high-resolutionoutput image.
 2. The system of claim 1, wherein the plurality ofgradient direction values are generated by the following steps:generating a mean difference between a neighbor sub-block and the centersub-block along a direction; and repeating the step of generating themean difference along other directions.
 3. The system of claim 2,wherein the mean differences for all the directions are quantized intoone of three quantization levels, thereby resulting in a ternary number.4. The system of claim 3, wherein the gradient magnitude value isgenerated by the following step: generating a root mean square of themean differences for all the directions to result in the gradientmagnitude value.
 5. The system of claim 1, wherein digits of thequantized gradient magnitude value are more significant than digits ofthe quantized gradient direction value.
 6. The system of claim 1,wherein the patch with the interpolated pixel is normalized before thepatch is processed by the clustering device.
 7. The system of claim 6,wherein the patch of the output image is de-normalized.
 8. The system ofclaim 1, wherein the patch with the enhanced pixel is fed back to gothrough the clustering device, the mapping device and the predictiondevice at least one time to generate an updated enhanced pixel,therefore refining the output image.
 9. The system of claim 8, whereineach iteration of the patch with the enhanced pixel being fed back to gothrough the clustering device, the mapping device and the predictiondevice is executed with a corresponding updated function table stored inthe mapping device.